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Agenda 

Objectives,  recent  and  current  work 
Research  on  CMMI®  Impact 
Characterizing  Impacts 

Benefits  of  CMMI-based  Process  Improvement 
Recently  reported  CMMI  and  CMM®  results 
Proposed  future  directions 


Carnegie  Mellon 

Software  Engineering  Institute 


Overall  Objectives 

Provide  credible,  objective  evidence  about 
organizations'  experiences  with  CMMI  based  process 
improvement. 

Focus: 

•  Impact  and  value  added 

•  Investment  and  costs  incurred 

•  Conditions  of  successful  adoption,  transition,  and 
documented  improvement 

•  Pitfalls  and  obstacles  to  successful  adoption  and  use 
Conduct  objective  studies  that  inform  the  development 
and  evolution  of  the  CMMI  product  suite 
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Recent  &  Current  Work 

Collaborative  case  studies 

•  Early  adopters  with  credible  quantitative  evidence  of  impact 
and  benefits  of  CMMI 

•  Selected  supplementary  evidence 
SEI  Special  Report 

•  Demonstrating  the  Impact  and  Benefits  of  CMMI®: 

An  Update  and  Preliminary  Results 

•  Based  on  case  studies,  supplementary  materials,  and 
comprehensive  literature  review 

Track  at  3rd  Annual  CMMI  Technology  Conference  and 
User  Group 

•  14  case  study  presentations  &  keystone  summary  presentation 

•  Roundtable  panel  with  discussion  of  next  steps 
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-  Carnegie  Mellon 

Software  Engineering  Institute 

Organizations 


CMMI  Conference  Presenters 

Others 

Accenture 

Boeing  Ltd,  Australia 

CMS  Information  Services,  Inc. 

Bosch  Gasoline  Systems 

Harris  Corporation 

Fort  Sill  Fire  Support  Software 

Engineering  Center 

Lockheed  Martin  Management  and  Data 

Systems 

General  Motors  Corporation 

Lockheed  Martin  Maritime  Systems  and 

Sensors  -  Undersea  Systems 

J.P.  Morgan  Chase  &  Co. 

Lockheed  Martin  Systems  Integration 

Sanchez  Computer  Associates,  Inc. 

Motorola  Global  Software  Group,  India 

Thales  Air  Traffic  Management 

Northrop  Grumman  Defense  Enterprise 

Systems 

Thales  Research  &  Technology 

Raytheon  North  Texas  Software  Engineering 

Thales  Training  &  Simulation 

Siemens  Information  Systems  Ltd,  India 

Plus  2  Anonymous 
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Research  on  CMMI  Impact 

Objectives,  recent  and  current  work 
Research  on  CMMI®  Impact 
Characterizing  Impacts 

Benefits  of  CMMI-based  Process  Improvement 
Recently  reported  CMMI  and  CMM®  results 
Proposed  future  directions 
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Why  Do  We  Need  Objective 
Evidence? 

Increasing  numbers  of  organizations  are  considering  using 
CMMI  models 

Trustworthy  evidence  is  essential  for 

•  Addressing  skepticism  about  model-based  process 
improvement  in  general 

•  Demonstrate  the  value  of  CMMI  over  its  source  models 

•  Building  commitment  and  obtaining  resources  within  an 
organization 

•  Enhancing  ongoing  quantitative  management 

•  Providing  input  for  improving  organizational  processes  and 
technologies 

•  Comparing  results  with  those  of  comparable  organizations 
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What  is  Legitimate  Evidence  of 
Impact? 

Evidence  based  on: 

•  New  processes  or  changes  to  existing  processes  due 
to  CMMI 

•  Broadened  organizational  scope  across  disciplines 

-  Especially  for  software  intensive  systems 

•  Process  changes  that  are  consistent  with,  but 
predate,  CMMI 

-  Especially  in  organizations  appraised  early  at 
higher  CMMI  maturity  levels 

•  Recent  evidence  based  on  the  SW-CMM 

-  Much  of  the  same  content  is  present  in  CMMI 
models 

-  And,  such  evidence  can  be  compelling  to  skeptics 
about  any  CMM-based  process  improvement 
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Generalizability 

Case  studies 

•  Offer  a  great  deal  of  valuable  detail  and  context 

•  Provide  lessons  learned  which  can  be  used  to  guide 
future  improvement  efforts 

•  Demonstrate  what  can  happen  under  the  right 
organizational  and  technical  circumstances 

•  However,  results  from  individual  case  studies  cannot 
be  generalized 

Our  task  is  to  design  studies  that  better  reflect  the 
experiences  of  the  wider  CMMI  community 
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Characterizing  Impacts 

Objectives  ,  recent  and  current  work 
Research  on  CMMI®  Impact 
Characterizing  Impacts 

Benefits  of  CMMI-based  Process  Improvement 
Recently  reported  CMMI  and  CMM®  results 
Proposed  future  directions 
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Impacts:  Costs  and  Benefits  of  CMMI 


Process 

Capability  &  _ 
Organizational 
Maturity 


BENEFITS 

Process 

Adherence 

Cost 

Schedule 

Productivity 

Quality 

Customer 

Satisfaction 


page  11 


Carnegie  Mellon 

Software  Engineering  Institute 


Seven  Kinds  of  Performance 
Measures 

From  the  previous  set,  we  found  examples  of  7  different 
categories  of  performance  measures 

•  Process  Adherence 

•  Cost 

•  Schedule 

•  Productivity 

•  Quality 

•  Customer  Satisfaction 

•  Return  on  Investment 
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Impact  of  CMMI-Based  Process 
Improvement 

Objectives  and  review  current  work 
Research  on  CMMI®  Impact 
Characterizing  Impacts 
Impact  of  CMMI-based  Process  Improvement 
Recently  reported  CMMI  and  CMM®  results 
Proposed  future  directions 
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Impact:  Process  Adherence 
and  Cost  of  Quality 

•  Work  product  completion  improved  dramatically  (CMS 
Information  Services,  Inc.) 

•  Exceeded  goal  for  reduction  in  cost  of  poor  quality 
(Motorola  Global  Software  Group,  India) 

•  Improved  adherence  to  quantitative  management  practices 
(Raytheon  North  Texas  Software  Engineering) 

•  Reduced  cost  of  poor  quality  from  over  45  percent  to  under 
30  percent  (Siemens  Information  Systems  Ltd,  India) 

•  Used  Measurement  and  Analysis  to  significantly  reduce  the 
cost  of  quality  in  one  year  (reported  under  non  disclosure) 
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Impact:  Cost, 

•  33  percent  decrease  in  the  average  cost  to  fix  a  defect 
(Boeing,  Australia) 

•  20  percent  reduction  in  unit  software  costs  (Lockheed 
Martin  Management  and  Data  Systems) 

•  15  percent  decrease  in  defect  find  and  fix  costs 
(Lockheed  Martin  Management  and  Data  Systems) 

•  4.5  percent  decline  in  overhead  rate  (Lockheed  Martin 
Management  and  Data  Systems) 

•  Improved  and  stabilized  Cost  Performance  Index 
(Northrop  Grumman  Defense  Enterprise  Systems) 
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Impact:  Cost2 

•  Increased  accuracy  in  cost  estimation  (Raytheon  North 
Texas  Software  Engineering) 

•  5  percent  improvement  in  average  cost  performance 
index  with  a  decline  in  variation  (Raytheon  North  Texas 
Software  Engineering) 

-  As  the  organization  improved  from  SW-CMM  level  4  to 
CMMI  level  5 

•  $2.1  Million  in  savings  in  hardware  engineering  processes 
(reported  under  non  disclosure) 
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Impact:  Schedule., 

•  50%  reduction  in  release  turn  around  time  (Boeing, 
Australia) 

•  60  percent  reduction  in  work  and  fewer  outstanding 
actions  following  pre-test  and  post-test  audits  (Boeing, 
Australia) 

•  Increased  the  percentage  of  milestones  met  from 
approximately  50  percent  to  approximately  95  percent 
(General  Motors) 

•  Decreased  the  average  number  of  days  late  from 
approximately  50  to  fewer  than  10  (General  Motors) 

•  Increased  through-put  resulting  in  more  releases  per  year 
(JP  Morgan  Chase) 
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Impact:  Schedule2 

•  Improved  and  stabilized  Schedule  Performance  Index 
(Northrop  Grumman  Defense  Enterprise  Systems) 

•  Met  every  milestone  (25  in  a  row)  on  time,  with  high 
quality  and  customer  satisfaction  (Northrop  Grumman 
Defense  Enterprise  Systems) 

•  Reduced  variation  in  schedule  performance  index 
(Raytheon  North  Texas  Software  Engineering) 

•  Reduced  schedule  variance  over  20  percent  (reported 
under  non  disclosure) 

•  Achieved  95  percent  on  time  delivery  (reported  under  non 
disclosure) 


page  18 


Carnegie  Mellon 

Software  Engineering  Institute 


Impact:  Productivity 

•  Improved  productivity  substantially,  with  “significantly  more 
rigorous  engineering  practices”  due  to  CMMI  (Fort  Sill  Fire  Support 
Software  Engineering  Center) 

•  Increased  productivity  after  adoption  of  CMMI  (Harris  Corporation) 

•  30  percent  increase  in  software  productivity  (Lockheed  Martin 
Management  and  Data  Systems) 

•  Improved  software  productivity  (including  reuse)  from 
approximately  80  percent  in  1992  baseline  to  over  140  percent  at 
CMMI  ML  5  (Lockheed  Martin  Systems  Integration) 

•  25  percent  productivity  improvement  in  3  years  (Siemens 
Information  Systems  Ltd,  India) 

•  11  percent  increase  in  productivity,  corresponding  to  $4.4M  in 
additional  value  (reported  under  non  disclosure) 
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Impact:  Qual^ 

•  Reduced  software  defects  substantially,  with 
“significantly  more  rigorous  engineering  practices”  due 
to  CMMI 

(Fort  Sill  Fire  Support  Software  Engineering  Center) 

•  Substantial  decrease  in  code  defects  after  adoption  of 
CMMI  (Harris  Corporation) 

•  Reduced  software-defects-per-million-delivered-SLOC 
by  over  50  percent  compared  to  defects  prior  to  CMMI 
(Lockheed  Martin  Systems  Integration) 

•  Reduced  defect  rate  at  CMMI  ML5  approximately  one 
third  compared  to  performance  at  SW-CMM  ML5 
(Lockheed  Martin  Maritime  Systems  &  Sensors  - 
Undersea  Systems) 

•  Met  qoal  of  20  +/-  5  defects  per  KLOC  (Northrop 
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Impact:  Quality2 

•  Only  2  percent  of  all  defects  found  in  the  fielded 
system  (Northrop  Grumman  Defense  Enterprise 
Systems) 

•  Reduced  identified  defects  from  6.6  per  KLOC  to  2.1 
over  5  causal  analysis  cycles  (Northrop  Grumman 
Defense  Enterprise  Systems) 

•  Increased  focus  on  quality  by  developers  (Northrop 
Grumman  Defense  Enterprise  Systems) 

•  Improved  defect  removal  before  test  from  50  percent 
to  70  percent,  leaving  0.35  post  release  defects  per 
KLOC  (Siemens  Information  Systems  Ltd,  India) 

•  44  percent  defect  reduction  following  causal  analysis 
cycle  at  maturity  level  2  (reported  under  non 
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Impact:  Customer  Satisfaction 

•  Increased  award  fees  by  55  percent  compared  to  an 
earlier  SW-CMM  baseline  at  maturity  level  2  (Lockheed 
Martin  Management  and  Data  Systems) 

•  Received  more  than  98  percent  of  possible  customer 
award  fees  (Northrop  Grumman  Defense  Enterprise 
Systems) 

•  Earned  a  rating  of  “Exceptional”  in  every  applicable 
category  on  their  Contractor  Performance  Evaluation 
Survey  (Northrop  Grumman  Defense  Enterprise  Systems) 

•  Improved  average  customer  satisfaction  rating  10  percent 
(Siemens  Information  Systems  Ltd,  India) 
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Impact:  Return  on  Investment 

•  5:1  ROI  for  quality  activities  (Accenture) 

•  13:1  ROI  calculated  as  defects  avoided  per  hour  spent  in 
training  and  defect  prevention  (Northrop  Grumman  Defense 
Enterprise  Systems) 

•  Avoided  $3.72M  in  costs  due  to  better  cost  performance 
(Raytheon  North  Texas  Software  Engineering) 

-  As  the  organization  improved  from  SW-CMM  level  4  to 
CMMI  level  5 

•  2:1  ROI  over  3  years  (Siemens  Information  Systems  Ltd, 
India) 

•  Processes  for  earlier  defect  detection,  improved  risk 
management,  and  better  project  control  implemented  after 
showing  positive  return  on  investment  during  pilot  (Thales 
TT&S) 

•  2.5:1  ROI  over  1st  year,  with  benefits  amortized  over  less 
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Recently  Report  CMMI  &  CMM 
Results 

Objectives  ,  recent  and  current  work 
Research  on  CMMI®  Impact 
Characterizing  Impacts 

Benefits  of  CMMI-based  Process  Improvement 
Recently  reported  CMMI  and  CMM®  results 
Proposed  future  directions 
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Selected  CMMI  Results 
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Lockheed  Martin  M&DS 


SW  CMM  ML2  (1993)  to  ML  3  (1996)  to  CMMI  ML5 

(2002) 


Improvements  in: 


Results 

•  captured  a  greater  percentage  of  available  award 
fees,  now  receiving  55  percent  more  compared  to 
the  baseline  that  remained  unrealized  at  SW-CMM 
level  2 

1996-2002 

•  Increased  software  productivity  by  30% 

•  Decreased  unit  software  cost  by  20% 

•  Decreased  defect  find  and  fix  costs  by  15% 


Customer 

satisfaction 


Productivity 
Product  cost 


Proprietary  sources  with  permission;  August  2003. 
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Northrop  Grumman  IT-1 

Defect  prevention  using  PSP  and  CAR  at  CMMI  ML5 


Integrating  PSPsm  and  CMMI®  Level  5.  Gabriel  Hoffman,  Northrop  Grumman  IT  .  May  1, 
2003. 
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Northrop  Grumman  IT-2 


Appraised  at  CMMI  ML  5  in  December  2002 


Improvements  in: 


Results 

•  met  25+  milestones  in  a  row 

•  earned  a  rating  of  “Exceptional”  in  every 
applicable  category  on  a  formal  Contractor 
Performance  Evaluation  Survey 


Schedule  / 
cycle  time 


Customer 

satisfaction 


Hours  Invested:  124  in  Defect  Prevention  (CAR) 
Hours  saved:  1650  hours  (15  hours  per 

defect) 

ROI:  13:1 


Quality 


Cost  of 
quality 
/ROI 


Integrating  PSPsm  and  CMMI®  Level  5.  Gabriel  Hoffman,  Northrop 
Grumman  IT  .  May  1,  2003 
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Accenture 

Transition  SW-CMM  to  CMMI  ML  3 

•  May  2001  to  May  2002 

•  Transition  Time:  1149  person  hours 

Key  Content 

Measurement  and  Analysis 
DAFL*  TS,  RM,  Change  Control 
1PPQ*  visions,  OEI 
Generic  Goals 

Results 

•  ROI:  5:1  (for  quality  activities) 


Costs: 


Improvements  in: 

Cost  of 
quality 
/ROI 


Innovation  Delivered.  CMMI®  Level  3  in  a  Large  Multi-Disciplinary 
Services  Organization.  Bengzon,  SEPG  2003 
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General  Motors  Corporation 


CMMI  focus  2001 

Goal  is  Integration  of  Supplier  Work  and  GM  Project 
Execution 


Results: 

•  Improved  schedule  -  projects  met  milestones  Improvements  in: 
and  were  fewer  days  late 

Schedule  / 
cycle  time 


I  MilcMonn  Mci] 


(  \  2IHI0  (  \  20111  I’url  200’ 


B.Ue  Pan  I  .ate 


CY  2000  CY  2001  Pari  2002 


Camping  on  a  Seesaw:  GM’s  IS&S  Process  Improvement  Approach. 

Hoffman,  Moore  &  Schatz,  SEPG  2003. 
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Boeing  Ltd,  Australia 


Making  transition  to  CMMI  from  SW-CMM 
and  EIA  731;  early  CMMI  pilot  in  Australia 

Improvements  in: 


RESULTS  on  One  Project 

•  33%  decrease  in  the  average  cost  to  fix  a 
defect 

•  Turnaround  time  for  releases  cut  in  half 

•  60%  reduction  in  work  from  Pre-Test  and 
Post-Test  Audits;  passed  with  few 
outstanding  actions 


Product  cost 


Schedule  /  cycle 
time 


Quality 


•  Increased  focus  on  product  quality 

•  Increased  focus  on  eliminating  defects 

Boeing  Australia,  Software 

EngQippj^^^jig^a  2003  conference. 
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Thales  ATM 


CMMI  Level  4  helps  THALES  meet  their 
business  objectives. 

•  Ability  to  see  into  the  future  with  a  known 
level  of  confidence 


•  Increasing  number  of  processes  under 
statistical  control 

•  Measurement  based  process  improvement 

•  Return  on  investment  due  to 

-  earlier  defect  detection 


Improvements  in: 


Quality 

Predictability 


-  improved  risk  management 

-  better  control  of  projects 

CMMI®  Level  4  Preparation:  The  Story  of  the  Chicken  and  the  Egg.  Anne 
De  Goeyse  and  Anne  Sophie  Luce,  Thales  ATM;  and  Annie  Kuntzmann- 
Combelles,  Q-Labs  France,  ESEPG  2003. 
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Thales  Training  &  Simulation 

•  Began  process  improvement  with  SW-CMM 
in  1992;  Level  3  achieved  in  1996 

•  Refocused  on  CMMI  to  broaden  effort  to 
systems  engineering 


Lessons  Learned: 

-  quarterly  internal  “CBA  IPI  like” 
assessments  measure  progress  and  help 
avoid  regression 

-  experience  gained  during 
implementation  of  SW-CMM  was  a  key 
factor  in  CMMI  success 

-  data  collected  on  software  has  shown 
decreases  in  project  cost  and  schedule 
variances  as  maturity  increased 


Improvements  in: 


Achieving  CMMI  level  2:  Keys  to  success.  Robert  Richard.  ESEPG  2003. 
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Recent  CMM®  (&  CMMI)  Results 
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Thales  Research  &  Technology 


CMM  data  from  another  Thales  Unit  used  by  Thales 
Research  &  Technology  as  part  of  rationale  to  begin  PI 
with  CMMI. 


Improvements  in 


ResuiEts  measured  by  a  THALES  Unit  0 


Average  CotI  valance  ■ 
reduced  &y  2 

On  au^-aga  Tale  to 
aECEptanoc!  -■  div  dcd  fcy 
24 

Number  d1  :elc*:l6 
during  Ouflnmer 
acceptance  -  iJtvufcd  ty 
K 

OdeC  uT  ;u&l3Ticr 
acceptance  ■■  reduced  frf 
&TS. 


for  Software  impact  on  cost  deviation 


Between  CHM  level  1  end  CWU  level  1 
1>]  aeftwart  stefum  mar  eaah  plaL 


*2  LCVul  1 


T‘ia!ej  ^.eseardi  4  T cell y  LK 


%  CoeI  dnMialicin 

rHALES 


Product 

cost 

Schedule  / 
cycle  time 

Quality 

Customer 

satisfaction 


Getting  Started  with  Process  Improvement  Using  the  CMMI®-  Carol  Marsh, 
Patrick  Vigier.  ESEPG  2003. 
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Bosch  Gasoline  Systems 

CMM  based  improvements 

•  Predictability  --  Internal  On-Time  Delivery 
improved 

by  15% 

•  Less  Rework  -  first  pass  yield  improved  by  10% 

•  Product  Quality  -  reduction  in  error  cases  in  the 
factory  by  one  order  of  magnitude 


Improvements  in: 

Schedule  /  cycle 
time 


Predictability 
-  Rework 


Quality 


Next  Steps  include 

•  Move  to  CMMI  and  applying  it  to  software,  system  and 
hardware 

•  Expand  process  improvement  program  to  include 
sales,  hardware  and  component  development 


Critical  success  factors  for  improvement  in  a  large  embedded  systems 
organisation.  Wolfgang  Stolz,  Robert  Bosch  GmbH  Gasoline  Systems  GS- 

EC/ESP  and  Hans-Jiirgen  Kugler,  Q-Labs  Software  Engineering,  ESEPG 
onm 
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Sanchez  Computer  Associates,  Inc. 


CMM  Level  1  to  Level  3  in  15  months.  6 
Months  later, 

•  saved  $2  million  in  first  6  months,  most 
through  early  detection  and  removal  of 
defects 


Improvements  in: 
Quality 


+ 

REVENUE 
/  SAVINGS 


In  addition, 

•  improved  quality  of  code 

•  robust  training  program 

•  applicability  of  process  outside  of  software 
programming 

Financial  Services  Software  Developer  Saves  $2  Million  in  Six  Months  with 
CMM®  Implementation.  David  Consulting  Group,  News  Release. 
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J.P.  Morgan  Chase  &  Co 


1st  CMM  success  2001 

today,  28  teams  at  CMM  Level  2 
CMMI  success  -  1st  team  ML3  in  2003 

Investment  in  PI  =  $4  million 


Results: 

•  Improved  predictability  of  delivery  schedule 

•  Reduction  of  post-release  defects 

•  Reduced  severity  of  post-release  defects 


Improvements  in: 


And,  from  CMMI  specifically 
•  Increased  through-put  =  more  releases  per  year 

Goal  to  achieve  CMMI  throughout  organization 


Predictability 

Quality 

Schedule  / 
cycle  time 

With  permission  from  presentation  to  the  SEI,  September  2003. 
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Proposed  Future  Directions 

Objectives  ,  recent  and  current  work 
Research  on  CMMI®  Impact 
Characterizing  Impacts 

Benefits  of  CMMI-based  Process  Improvement 
Recently  reported  CMMI  and  CMM®  results 
Proposed  future  directions 
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Proposed  for  FY2004  and  Beyond1 


Impact  and  benefits  of  systems  engineering 

•  Processes  with  heritage  in  EIA  731  and  precursors 

•  Organizational  integration 

Additional  case  studies 

•  In-depth  collaboration  with  the  SEI 

•  Self  reported  via  the  SEIR 

Broadly  based  studies 

•  State-of-the-practice  surveys  of  CMMI  impact  and  transition 

•  Analyses  using  existing  commercial  databases 

•  Community  benchmarking  of  process  and  performance 
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Proposed  for  FY2004  and  Beyond2 

Related  studies 

•  Research  and  development  on  costs  and  benefits  of  CMMI 
appraisal  methods 

•  Guidance  on  calculating  cost-benefit,  cost  effectiveness,  ROI, 
and  cost  of  quality 

•  CMMI  adoption  and  impact  in  small  and  medium  enterprises 
Decision  support 

•  Proactive  guidance  for  Decision  Analysis  and  Resolution 

•  Combining  computer  modeling  and  simulation  with 
empirical  results 

•  Validating  predictions  empirically 
Guidance  on  using  measurement  effectively 

Technical  Report,  conference  presentations  and  journal  articles 
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Contact  Information 

For  more  information  or  to  discuss  participation,  contact 

Dennis  R.  Goldenson 
dg@sei.cmu.edu 

Diane  L.  Gibson 
dlg@sei.cmu.edu 

Robert  W.  Ferguson 
rwf@sei.cmu.edu 
Software  Engineering  Institute 
Pittsburgh,  PA  15213 


